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ABSTRACT 

Denial of Service (DoS) attacks frequently happen on the Inter- 
net, paralyzing Internet services and causing millions of dollars 
of financial loss. This work presents NetFence, a scalable DoS- 
resistant network architecture. NetFence uses a novel mechanism, 
secure congestion policing feedback, to enable robust congestion 
policing inside the network. Bottleneck routers update the feed- 
back in packet headers to signal congestion, and access routers use 
it to police senders' traffic. Targeted DoS victims can use the secure 
congestion policing feedback as capability tokens to suppress un- 
wanted traffic. When compromised senders and receivers organize 
into pairs to congest a network link, NetFence provably guaran- 
tees a legitimate sender its fair share of network resources without 
keeping per-host state at the congested link. We use a Linux imple- 
mentation, ns-2 simulations, and theoretical analysis to show that 
NetFence is an effective and scalable DoS solution: it reduces the 
amount of state maintained by a congested router from per-host to 
at most per-(Autonomous System). 

1. INTRODUCTION 

Large-scale Denial of Service (DoS) attacks remain as a po- 
tent threat to the Internet. A survey from Arbor Networks shows 
that DoS attacks continue to grow in both scale and sophistica- 
tion [4]. The largest observed attack reached 49Gbps in 2009, a 
104% growth over the past two years. The survey also ranks DoS 
attacks as the largest anticipated threat in the next 12 months. This 
result is not surprising, as tens of gigabits flooding traffic could 
easily overwhelm most links, routers, or sites on the Internet. 

The destructive nature of DoS attacks has brought forth a fun- 
damental research challenge: how can we design an open network 
architecture that is resistant to large-scale DoS attacks? There have 
been several proposals addressing this challenge [3,5,27,34,46,47]. 
These proposals enable DoS victims to suppress attack traffic using 
network capabilities or filters, but when malicious sender-receiver 
pairs collude to flood a link, the best defense mechanism these sys- 
tems can offer is per-host queuing at the flooded link to separate 
legitimate traffic from attack traffic. This solution faces a scalabil- 
ity challenge, as a flooded router may forward packets for millions 
of (malicious and legitimate) end systems. 

This paper presents the design and evaluation of NetFence, a 
scalable DoS-resistant network architecture. NetFence provably 
guarantees each sender its fair share of bandwidth without keep- 
ing per-host state at bottleneck routers even when malicious senders 
and receivers collude into pairs to flood the network. It also enables 
DoS victims to suppress unwanted traffic as in a capability-based 
system [34,47]. A key departure of NetFence from previous work 
is that it places the network at the first line of DoS defense rather 



than relies on end systems (be it senders or receivers) to suppress 
attack traffic. 

The NetFence design places a robust traffic policing control loop 
inside the network (§ 3 and § 4). Packets carry unforgeable con- 
gestion policing feedback stamped by routers that suffer excessive 
congestion (caused either by DoS attacks or other reasons, which 
NetFence does not distinguish). Access routers at the trust bound- 
aries between the network and end systems examine the feedback 
and police the senders' traffic. A malicious sender cannot gain 
more than its fair share of bandwidth even if it colludes with a 
compromised receiver, because it cannot spoof valid congestion 
policing feedback. Innocent DoS victims can use the unforgeable 
congestion policing feedback as capability tokens to suppress the 
bulk of unwanted traffic, by not returning the feedback to mali- 
cious senders. To be fail-safe in case access routers are compro- 
mised, NetFence uses Autonomous System (AS)-level queues (or 
rate-limiters) to separate traffic from different source ASes, limit- 
ing DoS damage to the ASes that harbor the compromised routers. 

We have implemented NetFence in Linux and evaluated its over- 
head and performance using theoretical analysis (§ 3.4), testbed 
experiments, and large-scale simulations (§ 6). Our analysis shows 
that regardless of attackers' strategies, NetFence provides a legit- 
imate sender its fair share of bottleneck bandwidth. The simula- 
tion results correlate well with this analysis, and also show that 
NetFence performs similarly to state-of-the-art capability- or filter- 
plus-fair-queuing DoS defense systems [27,47]. Our Linux pro- 
totype benchmarking results show that NetFence's per-packet pro- 
cessing overhead is low. 

These results suggest that NetFence is an effective and scalable 
DoS solution. NetFence's bottleneck routers have O(l) per-packet 
computational overhead, and maintain at most per- AS state (more 
scalable design alternatives exist as discussed in § 4.5), while pre- 
vious work requires these bottleneck routers to keep per-host state 
to protect legitimate traffic. One concern for the NetFence design is 
that access routers need to keep per-(sender, bottleneck link) state 
(§ 3), but we show in § 5.1 today's access routers can meet such 
scalability requirements. 

The key contributions of this paper include a new DoS defense 
primitive: secure congestion policing feedback, and based on it, 
the construction of a robust, network-based, closed-loop conges- 
tion policing architecture that scalably and effectively limits the 
damage of DoS flooding attacks. With a closed-loop design, Net- 
Fence can flexibly place different functionalities at different lo- 
cations: lightweight attack detection and congestion signaling at 
bottleneck links, and congestion policing that requires per-(sender, 
bottleneck link) state at access routers. This design makes it scale 
much better than previous open-loop approaches that employ per- 
host queuing at bottleneck routers [27,47]. 



2. ASSUMPTIONS AND GOALS 

Before we present the design of NetFence, we first describe its 
threat model, assumptions, and design goals. 

2.1 Threat Model and Assumptions 

Flood-based network attacks: NetFence focuses on mitigating 
network-layer flooding attacks where attackers send excessive traf- 
fic to exhaust network resources such as link capacity or router pro- 
cessing power. It does not aim to mitigate DoS attacks that exploit 
application vulnerabilities to exhaust end system resources. 

Strong adversary: We assume that attackers can compromise 
both end systems and routers. Compromised end systems involved 
in an attack can grow into millions; they may launch brute-force or 
strategic flooding attacks. For instance, they may disguise attack 
traffic as legitimate traffic, launch on-off attacks, or collude into 
sender-receiver pairs to send flooding traffic. Attack traffic may or 
may not be distinguishable from legitimate traffic. 

We make two assumptions to assist NetFence's design. 

Trust: We assume that routers managed by the network are much 
less likely to be compromised than end systems. We thus place 
policing functions on routers rather than end systems. As a tradeoff 
for scalability, we treat each AS as a trust and fate sharing unit. 
When compromised routers exist, we aim to localize the damage 
to the ASes that harbor compromised routers rather than protect all 
the legitimate hosts within such ASes. 

Line-speed lightweight cryptography: We assume that symmet- 
ric key cryptography can be supported at line-speed. Some current 
hardware can support AES operations at 40Gbps [20], and the latest 
Intel Westmere processors have native support for AES [21]. 

2.2 Goals 

NetFence aims to meet several design goals. It is these goals that 
distinguish NetFence from previous work. 

i) Guaranteed network resource fair share: When DoS victims 
can identify attack traffic, we aim to enable them to suppress the at- 
tack traffic near the origins. This prevents attack traffic from wast- 
ing network resources. When DoS victims fail to identify attack 
traffic, or when attackers collude into sender-receiver pairs to flood 
the network, we resort to a weaker goal to guarantee a legitimate 
sender its fair share of network resources. That is, for any link of 
capacity C shared by N (legitimate and malicious) senders, each 
sender with sufficient demand should be guaranteed at least O(^) 
bandwidth share from that link. This mitigates the effect of large- 
scale DoS attacks from denial of service to predictable delay of 
service. 

ii) Open network: NetFence aims to keep the network open to 
new applications, and thus places the attack traffic identification 
function at the receivers to avoid false positives introduced by in- 
network traffic classification. This goal is also shared by previous 
work [3,5,47]. 

iii) Scalable and lightweight: NetFence may face millions of at- 
tackers that attempt to congest a single link in the network. To be 
effective at such a scale, it does not assume that a router always 
has sufficient resources to warrant per-flow or per-host state man- 
agement. It aims to keep little or no state in the core network and 
avoid heavyweight operations such as per-flow/host fair queuing 
in the core network. To facilitate high-speed router implementa- 
tion, NetFence aims to incur low communication, computation, and 
memory overhead. 

iv) Robust: NetFence should be robust against both simple, brute- 
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Figure 1: The NetFence architecture. Packets carry unspoofable con- 
gestion policing feedback stamped by bottleneck routers in this 
figure). Access routers (R a ) use the feedback to police senders' traffic, 
preventing malicious senders from gaining unfair shares of bottleneck 
capacity. DoS victims can use the congestion policing feedback as ca- 
pability tokens to suppress unwanted traffic. 

force flooding attacks and sophisticated ones that attempt to bypass 
or abuse NetFence itself. 

v) Incrementally adoptable: We aim to make NetFence incre- 
mentally deployable on today's Internet. Specifically, we aim to 
provide early adopters immediate deployment benefits: they can 
form an "overlay" network of deployed regions and benefit col- 
lectively from the deployment. We aim not to require hop-by-hop 
deployment from a congested link to compromised end systems to 
be effective, unlike [29]. 

vi) Network self-reliant defense: We aim for a self-reliant solu- 
tion that depends on only routers in the network, not other infras- 
tructures such as trusted host hardware [2] or DNS extensions [34]. 
Our hypothesis is that extra dependencies increase security risk and 
may create deployment deadlocks. That is, without the deploy- 
ment or upgrade of other infrastructures, the design is not effective. 
Hence, there is little incentive to deploy it, and vice versa. 

3. ARCHITECTURE 

In this section, we present an overview of the NetFence architec- 
ture, and defer design details to § 4. 

3.1 System Components 

NetFence has three types of packets: request packets, regular 
packets, and legacy packets. The first two, identified by a special 
protocol number in the IP header, have a shim NetFence header 
between their IP and upper-layer protocol headers. The NetFence 
header carries unforgeable congestion policing feedback generated 
by the network (§ 3.2 and § 4.4). A NetFence-ready sender sends 
request and regular packets, while a non-NetFence sender sends 
only legacy packets. 

Each NetFence router, depicted in Figure 2, keeps three chan- 
nels, one for each of the three packet types discussed above. To 
motivate end systems to upgrade, the NetFence design gives legacy 
channel lower forwarding priority than the other two. To prevent re- 
quest flooding attacks from denying legitimate requests, NetFence 
has a priority-based backoff mechanism for the request channel 
(§ 4.2). The request channel is also limited to consume no more 
than a small fraction (5%) of the output link capacity, as in [34,47]. 

NetFence places its feedback and policing functions at bottle- 
neck and access routers that are either inside the network or at the 
trust boundaries between the network and end systems. It does 
not place any trusted function at end systems. As shown in Fig- 
ure 1, a NetFence sender starts an end-to-end communication by 
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Figure 2: Each NetFence router keeps three channels. 



sending request packets to its NetFence-ready receiver (Step 1). 
The access router inserts the nop feedback in the NetFence header 
of the packet (Step 2, § 4.1). Along the path, a bottleneck router 
might modify the feedback, in a way similar to TCP ECN [36] (Step 
3). After the receiver returns the feedback to the sender (Step 4), 
the sender can send valid regular packets that contain the feedback 
(Step 5). In Step 4, two-way protocols like TCP can piggyback the 
returned feedback in their data packets, while one-way transport 
protocols such as UDP must send extra, low-rate feedback packets 
from a receiver to a sender. 

A NetFence router periodically examines each output link to de- 
cide if an attack is happening at the link. It uses a combination of 
link load and packet loss rate as an attack indicator (§ 4.3.1). If an 
attack is detected, NetFence starts a monitoring cycle, which lasts 
until i) no more attack is detected during the cycle, and ii) the cycle 
has existed for an extended period (typically a few hours) after the 
most recent attack is detected. During a monitoring cycle, the mon 
congestion policing feedback (containing the link ID I, an action 
field, etc.) is stamped into the NetFence header of all the passing 
request/regular packets (§ 4.3.2). The sender's regular packets must 
include this mon feedback to be considered valid, and they will be 
policed by the access router (Step 6, § 4.3.3). 

An access router maintains one rate limiter for every sender- 
bottleneck pair to limit a sender's regular traffic traversing a bottle- 
neck link. The router uses an Additive Increase and Multiplicative 
Decrease (AIMD) algorithm to control the rate limit: it keeps the 
rate limit constant within one pre-defined control interval (a few 
seconds); across control intervals, it either increases the rate limit 
additively or decreases it multiplicatively, depending on the partic- 
ular mon feedback it receives (§ 4.3.4). We use AIMD to control 
the rate limit because it has long been shown to converge onto ef- 
ficiency and fairness [11]. Other design choices exist; they have 
different cost-performance tradeoffs, and are discussed in § 7. 

When no attack is detected, a downstream router will not modify 
the nop feedback stamped by an access router. When the sender 
obtains the nop feedback and presents it back to its access router 
in a packet, the packet will not be rate-limited. That is, when no 
attack happens, NetFence stays in idle state. The overhead during 
such idle periods is low, because 1) the NetFence header is short 
(20 bytes) (§ 6.1); 2) the bottleneck attack detection mechanism 
only involves a packet counter and a queue sampler; and 3) an ac- 
cess router only needs to stamp and validate (not rate limit) the 
NetFence header for each packet. Only when an attack is detected 
at a bottleneck link, does NetFence activate its policing functions, 
which add additional processing overhead at bottleneck and access 
routers. We show the overhead benchmarking results in § 6.2. 

3.2 Unforgeable Congestion Policing Feedback 

Congestion policing feedback must be made unforgeable so that 
malicious nodes cannot evade NetFence's traffic policing functions. 
NetFence achieves this goal using efficient symmetric key cryptog- 
raphy. An access router inserts a periodically changing secret in a 



packet's NetFence header. A bottleneck router uses this secret to 
protect its congestion policing feedback, and then erases the secret. 
The access router, knowing the secret, can validate the returned 
feedback. We describe the details of this design in § 4.4, and dis- 
cuss how to limit the effect of compromised access routers in § 4.5. 

3.3 Congestion Feedback as Capability 

If a DoS victim can identify and desires to bar attack traffic, Net- 
Fence's unspoofable congestion policing feedback also serves as a 
capability token: a receiver can return no feedback to a malicious 
sender. Because the malicious sender cannot forge valid feedback, 
it cannot send valid regular packets. It can at most flood request 
packets to a destination, but an access router will use a priority- 
based policing scheme to strictly limit a sender's request traffic rate 
(§ 4.2). Alternatively, it can simply flood to congest its local area 
network, but this attack is easy to detect and the damage is confined 
to the local area network. 

3.4 Fair Share Guarantee 

With the above-described closed-loop network architecture, we 
are able to prove that NetFence achieves per-sender fairness for 
single bottleneck scenarios: 

Theorem: Given G legitimate and B malicious senders sharing 
a bottleneck link of capacity C, regardless of the attack strategies, 
any legitimate sender g with sufficient demand eventually obtains a 
capacity fair share -jfqrg-. where < v g < 1 is a parameter deter- 
mined by how efficient the sender g 's transport protocol ( e.g., TCP) 
utilizes the rate limit allocated to it, and p is a parameter close to 
1, determined by NetFence 's implementation-dependent AIMD and 
attack detection parameters. 

We briefly describe why this theorem holds, but leave a detailed 
proof in Appendix A. 

Proof sketch: In NetFence, an access router keeps one rate lim- 
iter for each sender-bottleneck pair when a monitoring cycle is 
triggered during attack times. Based on the unspoofable conges- 
tion feedback from the bottleneck, the access router dynamically 
adjusts the rate limits using a robust AIMD algorithm (§ 4.3.4). 
Since AIMD has been shown to converge onto efficiency and fair- 
ness [11], all the rate limits will eventually converge to the fair 
share of the bottleneck capacity. Thus, any sender, whether legiti- 
mate or malicious, can send at most as fast as its fair share rate. 

In the next section, we will show the design details that make 
NetFence achieve this lower bound on fairness despite various brute- 
force and strategic flooding attacks. 

4. DESIGN DETAILS 

In this section, we show the design details of NetFence. For 
clarity, we first present the design assuming unforgeable congestion 
policing feedback and non-compromised routers. We then describe 
how to make congestion policing feedback unforgeable and how to 
handle compromised routers. Key notations used to describe the 
design are summarized in Figure 3. 

4.1 Congestion Policing Feedback 

NetFence uses three types of congestion policing feedback: 

• nop, indicating no policing action is needed; 

• L^, indicating the link L is overloaded, and an access router 
should reduce the traffic traversing L; 

• , indicating the link L is underloaded, and an access router 
can allow more traffic traversing L. 

We refer to and as the mon feedback. Each congestion 
policing feedback includes a timestamp to indicate its freshness. 
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Figure 3: Key parameters and their values in our implementation. 



4.2 Protecting the Request Channel 

Attackers may simply flood request packets to congest down- 
stream links. NetFence mitigates this attack with two mechanisms. 
First, it limits the request channel on any link to a small fraction 
(5%) of the link's capacity, as in [34,47]. This prevents request 
packets from starving regular packets. Second, it combines packet 
prioritization and priority-based rate limiting to ensure that a legit- 
imate sender can always successfully transmit a request packet if it 
waits long enough to send the packet with high priority. This mech- 
anism ensures that a legitimate user can obtain the valid congestion 
policing feedback needed for sending regular packets. 

In NetFence, a sender can assign different priority levels to its 
request packets. Routers forward a level-fc packet with higher pri- 
ority than lower-level packets, but the sender is limited to send 
level-fc packets at half of the rate of level-(fc-l) packets. An ac- 
cess router installs per-sender token-based rate limiters to impose 
this rate limit. It removes 2 fc ~ 1 tokens from a request packet rate 
limiter when admitting a level-fc packet. Level-0 packets are not 
rate-limited, but they have the lowest priority. 

This request channel policing algorithm guarantees that a legiti- 
mate sender can eventually send a request packet to a receiver re- 
gardless of the number of attackers [34]. It holds because the ar- 
rival rate of request packets decreases exponentially as their priority 
level increases. Thus, the arrival rate of high priority request pack- 
ets will eventually be smaller than the request channel capacity. 

NetFence does not use computational puzzles as in [34]. This is 
because computational resources may be scarce [13], especially in 
busy servers and handheld devices. In addition, NetFence's design 
has the flexibility that an access router can configure different token 
refill rates for different hosts on its subnet. Legitimate servers could 
be given a higher rate to send more high priority request packets 
without purchasing additional CPU power. 

When an access router forwards a request packet to the next hop, 
it stamps the nop feedback into the packet, ensuring that a sender 
can obtain valid feedback if the receiver desires to receive from it. 

4.3 Protecting the Regular Channel 

Malicious senders may flood regular packets when they can ob- 
tain valid congestion policing feedback from their colluding re- 
ceivers. We describe how to mitigate this attack. 

4.3.1 A Monitoring Cycle 

When a router suspects that its outgoing link L is under attack, 
it starts a monitoring cycle for L. That is, it marks L as in the 
mon state and starts updating the congestion policing feedback in 
packets that traverse L (§ 4.3.2). Once a sender's access router 
receives such feedback, it will start rate limiting the sender's regular 
packets that will traverse the link L (§ 4.3.3). 

It is difficult to detect if L is under an attack because the at- 



tack traffic may be indistinguishable from legitimate traffic. In Net- 
Fence, L's router infers an attack based on L's utilization and the 
loss rate of regular packets. If L is well-provisioned and its normal 
utilization is low (a common case in practice), it can be consid- 
ered as under an attack when its average utilization becomes high 
{e.g., 95%); if L always operates at or near full capacity, its router 
can infer an attack when the regular packets' average loss rate p 
exceeds a threshold p t h- A link's average utilization and p can be 
calculated using the standard Exponentially Weighted Moving Av- 
erage (EWMA) algorithm [18]. The threshold p t h is a local policy 
decision of L's router, but it should be sufficiently small so that 
loss-sensitive protocols such as TCP can function well when no at- 
tack is detected. Attackers may launch a mild attack and evade the 
detection by keeping p below p t h, but the damage is also limited. 

When the attack detection is based on the packet loss rate p, a 
flash crowd may also be considered as an attack. We do not distin- 
guish these two because it is too difficult to do so. As shown by our 
simulation results (§ 6), starting a monitoring cycle for a link does 
not have much negative impact on a legitimate sender. 

It is undesirable to infinitely keep a monitoring cycle due to the 
added overhead. Thus, a NetFence router terminates a link L's 
monitoring cycle when L is no longer under attack (e.g., p < p t h) 
for a sufficiently long period of time Tb- The router will mark L as 
in the nop state and stop updating the congestion policing feedback 
in packets traversing L. Similarly, an access router will terminate 
a rate limiter (sre, L) if it has not received any packet with the 
feedback and the rate limiter has not discarded any packet for T a 
seconds. 

Routers should set T a and TJ, to be significantly longer (e.g., a 
few hours) than the time it takes to detect an attack (Td). This is 
because attackers may flood the network again after T a (or Ti,) sec- 
onds. By increasing the ratio of the monitored period min(T a , Tb) 
to the unprotected period Td, we reduce the network disruption 
time. Network disruption during an attack detection period can- 
not be eliminated unless compromised senders are patched up, but 
we do not assume routers have this ability. 

4.3.2 Updating Congestion Policing Feedback 

When a link L is in the mon state, its router Rb uses the follow- 
ing ordered rules to update the congestion policing feedback in any 
request/regular packet traversing L: 

1. If the packet carries nop, stamp 

2. Otherwise, if the packet carries L'^ stamped by an upstream 
link L' , do nothing; 

3. Otherwise, if L is overloaded, stamp lA 

The router Rb never stamps the L^ feedback. As we will see in 
§ 4.3.3, only an access router stamps L^ when forwarding a packet. 
If the feedback reaches the receiver of the packet, it indicates 
that the link L is not overloaded, because otherwise the router Rb 
would replace the feedback with the feedback. 

A packet may cross multiple links in the mon state. The access 
router must ensure that the sender's rate does not exceed its legit- 
imate share at any of these links. The second rule above allows 
NetFence to achieve this goal, gradually. This is because the first 
link Li on a packet's forwarding path that is both overloaded and in 
the mon state can always stamp the L\ feedback, and downstream 
links will not overwrite it. When the L\ feedback is presented to 
an access router, the router will reduce the sender's rate limit for 
the link Li until Li is not overloaded and does not stamp L\. This 
would enable the next link (L2) on the path that is both in the mon 
state and overloaded to stamp Lj into the packets. Gradually, a 
sender's rate will be limited such that it does not exceed its fair 
share on any of the on-path links in the mon state. 
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Figure 4: Once a router Rb encounters congestion between time 
[t, ti], it will continuously stamp the feedback until t\ + 2In m . 



4.3.3 Regular Packet Policing at Access Routers 

A sender src's access router polices the sender's regular packets 
based on the congestion policing feedback in its packets. If a packet 
carries the nop feedback, indicating no downstream links require 
congestion policing, the packet will not be rate-limited. Otherwise, 
if it carries or L^, it must pass the rate limiter (src, L). 

We implement a rate limiter as a queue whose de-queuing rate 
is the rate limit, similar to a leaky bucket [43]. We use the queue 
to absorb traffic bursts so that bursty protocols such as TCP can 
function well with the rate limiter. We do not use a token bucket 
because it allows a sender to send short bursts at a speed exceeding 
its rate limit. Strategic attackers may synchronize their bursts to 
temporarily congest a link, leading to successful on-off attacks. 

When an access router forwards a regular packet to the next hop, 
it resets the congestion policing feedback. If the old feedback is 
nop, the access router refreshes the timestamp of the feedback. If 
the old feedback is or L^, the access router resets it to L' . This 
design reduces the computational overhead at the link L's router, 
as it does not update a packet's feedback if L is not overloaded. 

For simplicity, NetFence uses at most one rate limiter to police 
a regular packet. One potential problem is that a flow may switch 
between multiple rate limiters when its bottleneck link changes. We 
discuss this issue in § 4.3.5. 

4.3.4 Robust Rate Limit Adjustment 

The and L^ feedback enables an access router to adjust a 
rate limiter (src, L)'s rate limit r;; m with an AIMD algorithm. A 
strawman design would decrease m m multiplicatively if the link L 
is overloaded and stamps the L^ feedback, or increase it additively 
otherwise. However, a malicious sender can manipulate this design 
by hiding the L^ feedback to prevent its rate limit from decreasing. 

To address this problem, we periodically adjust a rate limit, use 
L' as a robust signal to increase the rate limit, and ensure that a 
sender cannot obtain valid feedback for a full control interval if 
its traffic congests the link L. Let Iu m denote the control interval 
length for rate adjustment on an access router. Suppose a down- 
stream bottleneck router Rb has a link L in the mon state. Rb 
monitors L's congestion status using a load-based [45] or a loss- 
based algorithm such as Random Early Detection (RED) [18]. If 
Rb detects congestion between time t and ti, it will stamp the 
feedback into all packets traversing L from time t until two control 
intervals after ti: ti + 2L™, even if it has considered the link not 
congested after ti. This hysteresis ensures that if a sender congests 
a link L during one control interval, it will only receive the L^ 
feedback in the following control interval, as shown in Figure 4. 

For each rate limiter (src, L), the access router R a keeps two 
state variables: t a and haslncr, to track the feedback it has re- 



ceived. The variable t s records the start time of the rate limiter's 
current control interval, and haslncr records whether the rate lim- 
iter has seen the L^ feedback with a timestamp newer than t s . 
At the end of each control interval, R a adjusts the rate limiter 
(src, L)'s rate limit ru m as follows: 

1. If haslncr is true, R a compares the throughput of the rate 
limiter with \ru m - If the former is larger, ri im will be in- 
creased by A; otherwise, rj im will remain unchanged. Check- 
ing the rate limiter's current throughput prevents a malicious 
sender from inflating its rate limit by sending slowly for a 
long period of time. 

2. Otherwise, R a will decrease ru m to (1 — 5)ru m . 

We discuss how to set the parameters A, 8, etc. in § 4.6. 

We now explain why this AIMD algorithm is robust, i.e., a ma- 
licious sender cannot gain unfair bandwidth share by hiding the 
feedback: if a sender has sent a packet when a link L suffers 
congestion, the sender's rate limit for L will be decreased. Sup- 
pose L's router Rb detects congestion and starts stamping the L 1 
feedback at time t, and let t e denote the finishing time of an ac- 
cess router's control interval that includes the time t, as shown in 
Figure 4. Rb will stamp the feedback between [t, ti + 2Iu m ]. 
Since t e 6 [t, t + Ium], a sender will only receive the feedback 
for packets sent during the control interval [t e , t e + Iu m ], because 
t e > t and t e + Ium < t\ + Hum . It can either present the 
feedback newer than t e to its access router, or present one older 
than t e , or not send a packet. All these actions will cause its rate 
limit to decrease according to the second rule above. 

A legitimate sender should always present L^ feedback to its 
access router as long as the feedback has not expired, even if it has 
received newer L^ feedback. This design makes a legitimate sender 
mimic an aggressive sender's strategy and ensures fairness among 
all senders. 

4.3.5 Handling Multiple Bottlenecks 

When a flow traverses multiple links in the mon state, the flow's 
access router will instantiate multiple per-(sender, bottleneck link) 
rate limiters for the sender. The present NetFence design sends a 
regular packet to only one rate limiter for simplicity, but it may 
overly limit a sender's sending rate in some cases. This is be- 
cause when a sender's packets carry the congestion policing feed- 
back from one of the bottleneck links, all other rate limiters stay 
idle. The sender's access router will reduce their rate limits, if 
they are idle for longer than a full control interval, as described 
above (§ 4.3.4). Consequently, the idle rate limiters' rate limits 
may become smaller than a sender's fair share rates at those bottle- 
neck links. When the bottleneck link carried in a sender's packets 
changes, the sender may obtain less than its fair share bandwidth 
at the new bottleneck initially, until its rate limit for the new bot- 
tleneck converges. If the bottleneck link in the packets changes 
frequently, it is possible that none of the rate limits converge, giv- 
ing the sender a throughput smaller than its fair share bandwidth at 
any of the bottleneck links. In addition, when the bottleneck links' 
rate limits differ greatly and a sender's packets switch among them 
frequently, it may be difficult for a transport protocol such as TCP 
to adjust a flow's sending rate to match the abruptly changing rate 
limit, further reducing a sender's throughput. 

We have considered various solutions to address this problem. 
One simple solution is to allow a packet to carry all feedback from 

'This inequation also indicates that 2Iu m is the minimal hysteresis 
to ensure robustness. If the router Rb stamps the L 1 feedback for 
shorter than 2Iu m after ti, an attacker may obtain the feedback 
in the interval [t e ,t E + L im ]. 



all the bottleneck links on its path. An access router can then pass 
the packet through all the on-path rate limiters, each receiving its 
own feedback and policing the packet independently. This solution 
requires a longer and variable-length NetFence header. Another 
one is for an access router to infer the on-path bottleneck links of 
a packet based on history information and send the packet through 
all the inferred rate limiters. 

We do not include these solutions in the core design for simplic- 
ity. The details of these solutions can be found in Appendix B. Sim- 
ulation results in § 6.3.2 and Appendix B suggest that NetFence's 
performance is acceptable. Thus, we consider it a worthy tradeoff 
to keep the design simple. 

4.4 Securing Congestion Policing Feedback 

Congestion policing feedback must be unforgeable. Malicious 
end systems should not be able to forge or tamper the feedback, 
and malicious routers should not be able to modify or remove the 
feedback stamped by other routers. The NetFence design uses effi- 
cient symmetric key cryptography to achieve these goals. 

Feedback format: A congestion policing feedback consists of 
five key fields as shown in Figure 5: mode, link, action, ts, 
and MAC. When the mode field is nop, it represents the nop 
feedback. When the mode field is mon, the link field indicates 
the identifier (an IP address) of the corresponding link L, and the 
action field indicates the detailed feedback: if action is incr (deer), 
it is the (L^) feedback. The ts field records a timestamp, and 
the MAC field holds a MAC signature that attests the feedback's 
integrity. 

In addition to the five key fields, a mon feedback also includes a 
field token nop . We explain the use of this field later in this section. 

Stamping nop feedback: When an access router stamps the nop 
feedback, it sets mode to nop, link to a null identifier link nu u, 
action to incr, ts to its local time, and uses a time- varying secret 
key K a known only to itself to compute the MAC: 

token n0 p = MACK a (src,dst,ts,link nu u,nop) (1) 

The MAC computation covers both the source and destination 
addresses to prevent an attacker from re-using valid nop feedback 
on a different connection. 

Stamping feedback: When an access router stamps the 
feedback, the mode field is already mon, and the link field already 
contains the link identifier L. The router sets action to incr and 
ts to its local time, and computes the MAC field using the secret 
key K a : 
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MAC 



token L f = MACk„, (sre, dst, ts, L, mon, incr) 



(2) 



The router also inserts a token nop as computed in Eq (1) into 
the token nop field. 

Stamping L^ feedback: When a link L's router Rb stamps the 
feedback, it sets mode to mon, link to L, action to deer, 
and computes a new MAC value using a secret key K a i shared 
between its AS and the sender's AS: 

token L ± = M AC k (sre, dst, ts, L, mon, deer, token n op) (3) 

The shared secret key K a i is established by piggybacking a dis- 
tributed Diffie-Hellman key exchange in BGP as in [26]. The router 
Rb includes token nop stamped by the sender's access router in its 
MAC computation, and erases it afterwards to prevent malicious 
downstream routers from overwriting its feedback. 

If Rb is an AS internal router that does not speak BGP, it may 
not know K a i. In this case, Rb can leave the MAC and tokennop 



Figure 5: The key congestion policing feedback fields. 



fields unmodified and let an egress border router of the AS update 
their values when the packet exits the AS. This design reduces the 
management overhead to distribute K ai to an internal router Rb- 

Validating feedback: When a source access router receives a reg- 
ular packet, it first validates the packet's congestion policing feed- 
back. If the feedback is invalid, the packet will be treated as a 
request packet and subject to per-sender request packet policing. 

A feedback is considered invalid if its ts field is more than w sec- 
onds older than the access router's local time t, low : \t now — ts\ > 
w, or if the MAC field has an invalid signature. The MAC field 
is validated using Eq (1) and Eq (2) for the nop and feedback, 
respectively. To validate L^ feedback, the access router first re- 
computes the tokennop using Eq (1), and then re-computes the 
MAC using Eq (3). The second step requires the access router to 
identify the link L's AS in order to determine the shared secret key 
K a i. We can use an IP-to-AS mapping tool [32] for this purpose, 
as the feedback includes the link L's IP address. 

4.5 Localizing Damage of Compromised Routers 

The NetFence design places enforcement functions that include 
feedback validation and traffic policing at the edge of the network 
to be scalable. However, if an access router is compromised, attack- 
ers in its subnet or itself may misbehave to congest the network. 
NetFence addresses this problem by localizing the damage to the 
compromised AS. If an AS has a compromised router, we consider 
the AS as compromised, and do not aim to provide guaranteed net- 
work access for that AS's legitimate traffic. 

A NetFence router can take several approaches to localize the 
damage of compromised ASes, if its congestion persists after it 
has started a monitoring cycle, a signal of malfunctioning access 
routers. One approach is to separate each source AS's traffic into 
different queues. This requires per-AS queuing. We think the over- 
head is affordable because the present Internet has only about 35K 
ASes [7]. We may replace per-AS queuing with per-AS rate lim- 
iting and set the rate limits by periodically computing each AS's 
max-min fair share bandwidth on the congested link as in [29]. 
Another more scalable approach is to use a heavy-hitter detection 
algorithm such as RED-PD [30] to detect and throttle high-rate 
source ASes. A heavy-hitter detection algorithm is suitable in this 
case because legitimate source ASes will continuously reduce their 
senders' traffic as long as they receive the L^ feedback. The de- 
tected high-rate ASes are likely to be the compromised ASes that 
do not slow down their senders. 

All these approaches require a router to correctly identify the 
source AS of a packet, which can be achieved using an IP-to-AS 
mapping tool if the source IP address of the packet is not spoofed. 
NetFence uses Passport [26] to prevent source address spoofing. A 
Passport header is inserted between IP and the NetFence header. 
Passport piggybacks a distributed Diffie-Hellman key exchange in 
the inter-domain routing system to enable each pair of ASes to 
share a secret key. A source AS uses a key it shares with an AS 
on the path to a destination to compute a secure MAC and inserts 
it into a packet's Passport header. Each AS on the path can ver- 
ify that a packet is originated from the source AS by validating the 
corresponding MAC. NetFence also uses Passport to establish the 
shared secret keys between ASes to secure the congestion policing 
feedback (§ 4.4). 



4.6 Parameter Settings 

Figure 3 summarizes the main parameters in the NetFence de- 
sign and their values used in our implementation. The level- 1 re- 
quest packets (7i) are rate limited at one per 1 ms. A request 
packet size is estimated as 92 bytes that includes a 40-byte TCP/IP 
header, a 28-byte NetFence header (Figure 6) and a 24-byte Pass- 
port header [26]. We set the control interval Iu m to 2 seconds, one 
order of magnitude larger than a typical RTT (< 200ms) on the 
Internet. This allows an end-to-end congestion control mechanism 
such as TCP to reach a sender's rate limit during one control inter- 
val. We do not further increase 7j im because a large control interval 
would slow down the rate limit convergence. 

The rate limit AI parameter A can be neither too small nor too 
large: a small A would lead to slow convergence to fairness; a 
large A may result in significant overshoot. We set A to 12Kbps 
because it works well for our targeted fair share rate range: 50Kbps 
~ 400Kbps. A legitimate sender may abort a connection if its send- 
ing rate is much lower than 50Kbps, and 400Kbps should provide 
reasonable performance for a legitimate sender during DoS flood- 
ing attacks. The rate limit MD parameter 5 is set to 0.1, a value 
much smaller than TCP's MD parameter 0.5. This is because a 
router may stamp the feedback for two control intervals longer 
than the congestion period (§ 4.3.4). 

We set the attack detection threshold p t h to 2%, since at this 
packet loss rate, a TCP flow with 200ms RTT and 1500B packets 
can obtain about 500Kbps throughput [33]. We set a link's maxi- 
mum queue length Qu m to 200ms x the link's capability. We use 
a loss-based algorithm RED to detect a link's congestion status. It 
is our future work to implement a load-based algorithm {e.g., [45]). 

5. ANALYSIS 

In this section, we analyze the scalability and security of Net- 
Fence, and discuss the incremental deployment issues. 

5.1 Scalability 

As a closed-loop design, NetFence can place different functions 
at different locations to provide per-sender fairness. It places per- 
sender traffic policing at access routers, and lightweight congestion 
detection, feedback stamping, and AS-level policing at bottleneck 
routers. In contrast, per-host fair queuing, an open-loop solution 
used in previous work [27,47], does not have this flexibility. Every 
bottleneck router must keep per-host queues to provide per-sender 
(or per-receiver) fairness. There are only 35K ASes on today's In- 
ternet [7], while the number of compromised hosts involved in a 
DoS attack could reach millions [17]. Thus, compared to per-host 
fair queuing, NetFence can significantly reduce the amount of state 
kept by a bottleneck router. 

However, NetFence access routers need to perform per-(sender, 
bottleneck link) rate limiting. Our calculation suggests that with 
today's hardware technology, they can afford to do so and will not 
become a new scaling bottleneck. While we do not have accu- 
rate data to estimate the number of bottlenecks a sender traverses 
during attack times, we think 100 links per legitimate sender is a 
reasonable upper bound. An access router can aggregate a sender's 
rate limiters by bottleneck links' prefixes if a sender needs more 
than 100 rate limiters. If an access router serves 10K end hosts, it 
will have at most one million rate limiters in total. Each rate lim- 
iter needs about 24 bytes of memory for state variables (1 bit for 
haslncr, 8 bytes for two timestamps, 4 bytes for the rate limit, 
and 12 bytes for a queue object) and another 1500 bytes to queue at 
least one packet. The total amount of memory requirement is less 
than 2GB, and we can use fast DRAM for this purpose as access 
routers' line speeds are typically slower than those of core routers. 



The processing overhead of an access router is also acceptable. 
The per-packet processing time on our benchmarking PC is less 
than 1.3/j.s during attack times (§ 6.2). This translates into a through- 
put of 770K packets per second, or more than 9 Gbps, assuming 
1500-byte packet size and CPU is the throughput bottleneck. Im- 
plementing the cryptographic operations in hardware can further 
improve an access router's throughput. 

5.2 Security 

Next we summarize how NetFence withstands various attacks. 

5. 2. 1 Malicious End Systems 

Forgery or Tampering: Malicious end systems may attempt to 
forge valid congestion policing feedback. But NetFence protects 
congestion policing feedback with MAC signatures. As long as 
the underlying MAC is secure, malicious end systems cannot spoof 
valid feedback. A malicious sender may selectively present L* or 
hide L* to its access router, but NetFence's robust AIMD algorithm 
(§ 4.3.4) prevents it from gaining a higher rate limit. 

Evading attack detection: Malicious end systems may attempt to 
prevent a congested router from starting a monitoring cycle. This 
attack is ineffective when a well-provisioned router uses high link 
utilization to detect attacks. When an under-provisioned router uses 
the packet loss rate to detect attacks, NetFence limits the damage 
of this attack with a low loss detection threshold p t h (§ 4.3.1). 

On-off attacks: Attackers may attempt to launch on-off attacks. In 
a macroscopic on-off attack, attackers may flood the network again 
after a congested router terminates a monitoring cycle. NetFence 
uses a prolonged monitor cycle (§ 4.3.1) to mitigate this attack. In 
a microscopic on-off attack, attackers may send traffic bursts with 
a short on-off cycle, attempting to congest the network with syn- 
chronized bursts, yet maintaining average sending rates lower than 
their rate limits. Our theoretical bound in § 3 and simulation results 
in § 6.3.2 both show that the shape of attack traffic cannot reduce 
a legitimate user's guaranteed bandwidth share, because a sender 
cannot send faster than its rate limit at any time (§ 4.3.3), and Net- 
Fence's robust rate limit adjustment algorithm (§ 4.3.4) prevents a 
sender from suddenly increasing its actual sending rate. 

5.2.2 Malicious On-path Routers 

A malicious router downstream to a congested link may attempt 
to remove or modify the feedback stamped by a congested router 
in order to hide upstream congestion. But such attempts will make 
the feedback invalid, because the router does not know the original 
value needed to compute a valid MAC (§ 4.4). 

A malicious on-path router may discard packets to completely 
disrupt end-to-end communications, duplicate packets, or increase 
packet sizes to congest downstream links. It may also change the 
request packet priority field in a NetFence header to congest the 
request channel on downstream links. Preventing such attacks re- 
quires Byzantine tolerant routing [35], which is not NetFence's de- 
sign goal. Instead, we aim to make these attacks detectable. Pass- 
port [26], the source authentication system NetFence uses, partially 
protects the integrity of a packet and enables duplicate detection. It 
includes a packet's length and the first 8 bytes of a packet's trans- 
port payload (which includes the TCP/UDP checksum) in its MAC 
computation. We can further extend Passport's MAC computation 
to include NetFence's request packet priority field to protect it. 

5.3 Incremental Deployment 

NetFence can be incrementally deployed by end systems and 
routers. Since the NetFence header is a shim layer between IP 
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Figure 6: The NetFence header format. 



and upper layer protocols, legacy applications need not be modi- 
fied. Legacy routers can ignore the NetFence header and forward 
packets using the IP header. Routers at congested links and ac- 
cess routers need to be upgraded, but well-provisioned routers that 
can withstand tens of Gbps attack traffic may not need to upgrade. 
The deployment can take a bump-in-fhe-wire approach, by placing 
inline boxes that implement NetFence's enforcement functions in 
front of the routers that require upgrading. Middleboxes such as 
firewalls need to be configured to permit NetFence traffic. 

NetFence provides deployment incentives to both end systems 
and ASes, because legacy traffic is treated by deployed ASes with 
lower priority ( Figure 2). Deployed ASes can form a trusted over- 
lay network and protect each other's legitimate traffic within their 
networks. Their traffic is not protected at undeployed networks, en- 
couraging them to direct traffic to other deployed ASes using BGP. 

6. IMPLEMENTATION AND EVALUATION 

We have implemented NetFence prototypes in Linux and in the 
ns-2 simulator. Next we evaluate the NetFence header and packet 
processing overhead with our Linux implementation, and use ns-2 
simulations to show how effective NetFence mitigates DoS attacks. 

6.1 NetFence Header 

Figure 6 shows the format of a NetFence header in our Linux im- 
plementation. A full NetFence header from a sender to a receiver 
includes a forward header and a return header. The forward header 
includes the congestion policing feedback on the forward path from 
the sender to the receiver, and the return header includes the reverse 
path information from the receiver to the sender. Most fields are 
self-explained. A NetFence header is implemented as a shim layer 
between IP and an upper-layer protocol, and the PROTO field de- 
scribes the upper-layer protocol (e.g., TCP or UDP). The unit of a 
timestamp is one second. 

The return header may be omitted to reduce overhead if the sender 
has previously returned the latest feedback to the receiver. Even 
when the return header is present, it does not always include all the 
fields. If the returned feedback is nop, the LINK-ID re t urn field 
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Figure 7: NetFence implementation micro-benchmarking results. 

will be omitted because it is zero, and one bit in the FLAGS field 
indicates this omission. 

A NetFence header only includes the last two bits of the returned 
timestamp to save header space. In the subsequent packets from the 
sender to the receiver, the sender's access router will reconstruct 
the full timestamp from its local time and the returned two bits, 
assuming that the timestamp is less than four seconds older than 
its current time. With this implementation, a NetFence header is 
20 bytes in the common case when the feedback is nop for both 
the forward and return paths. In the worst case that the feedback is 
mon for both paths, the header is 28 bytes long. 

6.2 Micro-benchmarking 

We have implemented NetFence in Linux using XORP [19] and 
Click [24]. We modified XORP's BGP module to establish the 
pairwise symmetric keys shared between ASes. We added the data 
packet processing logic into Click and ran Click routers in the ker- 
nel space for packet forwarding. XORP communicates with Click 
via the /click file system. We added a module between the IP and 
transport layers on end-hosts to handle NetFence headers. This de- 
sign keeps upper-layer TCP/UDP protocols and legacy applications 
unmodified. We use AES-128 as a secure MAC function due to its 
fast speed and available hardware support [20,21]. 

We benchmark the Linux implementation on Deterlab [14] with 
a three-node testbed. A source access router A and a destination C 
are connected via a router B. The B — C link is the bottleneck with 
a capacity of 5Mbps. Each node has two Intel Xeon 3GHz CPUs 
and 2GB memory. To benchmark the processing overhead without 
attacks, we send 100Kbps UDP request packets and 1Mbps UDP 
regular packets from A to C respectively. To benchmark the over- 
head in face of DoS attacks, we send 1Mbps UDP request packets 
and 10Mbps UDP regular packets simultaneously. 

The benchmarking results are shown in Figure 7. With Net- 
Fence, when there is no attack, a request packet does not need any 
extra processing on the bottleneck router B, but it introduces an av- 
erage overhead of 546ns on the access router A because the router 
must stamp the nop feedback into the packet. A regular packet does 
not incur any extra processing overhead on the bottleneck router ei- 
ther, but it takes the access router 781ns on average to validate the 
returned feedback and generate a new one. When the bottleneck 
link enters the mon state during attack times, the bottleneck router 
takes 492ns to process a 92B request packet, or at most 554ns to 
process a 1500B regular packet. The access router takes on aver- 
age 1267ns to process a regular packet at attack times. 

The performance of a capability system TVA+ [27] on the same 
topology is also shown in Figure 7 for comparison. We can see 
that the processing overhead introduced by NetFence is on par with 
that of TVA+. Note that we do not show the result when TVA+ 
caches capabilities, because such caching requires per-flow state 
on routers, while NetFence does not have this requirement. 

These results show that NetFence's per-packet overhead is low. 
The CPU-intensive operations are primarily AES computation. Since 
there exists commercial hardware that can support AES operations 
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Figure 8: The average transfer time of a 20KB file when the targeted 
victim can identify and wish to remove the attack traffic. The file trans- 
fer completion ratio is 100% in all simulated systems. 

at 40Gbps [20], we expect that NetFence's per-packet processing 
will not become a performance bottleneck. We note that the bench- 
marking results do not include the Passport overhead, as a Passport 
header can be updated by inline boxes near an AS's ingress and 
egress border routers [26]. 

6.3 Mitigating DoS Flooding Attacks 

Next we evaluate how well NetFence mitigates various DoS flood- 
ing attacks using ns-2 simulations. We also compare NetFence with 
three other representative DoS mitigation schemes: 

TVA+ : TVA+ [27,47] is a network architecture that uses network 
capabilities and per-host fair queuing to defend against DoS flood- 
ing attacks. TVA+ uses two-level hierarchical fair queuing (first 
based on the source AS and then based on the source IP address) 
at congested links to mitigate request packet flooding attacks, and 
per-receiver fair queuing to mitigate authorized traffic flooding at- 
tacks in case (colluding or incompetent) receivers fail to stop attack 
traffic. 

Stoplt : Stoplt [27] is a filter and fair queuing based DoS defense 
system. A targeted victim can install network filters to stop un- 
wanted traffic. Similar to TVA+, in case receivers fail to install 
filters, Stoplt uses hierarchical queuing (first based on the source 
AS and then based on the source IP address) at congested links to 
separate legitimate traffic from attack traffic. 

Fair Queuing (FQ) : Per-sender fair queuing at every link provides 
a sender its fair share of the link's bandwidth. We use fair queuing 
to represent a DoS defense mechanism that aims to throttle attack 
traffic to consume no more than its fair share of bandwidth. 

We have implemented TVA+ and Stoplt as described in [27,47]. 
We use the Deficit Round Robin (DRR) algorithm [38] to imple- 
ment fair queuing because it has O(l) per packet operation over- 
head. In our simulations, attackers do not spoof source addresses 
because NetFence uses Passport [26] to prevent spoofing. Thus, 
routers could queue attack traffic separately from legitimate traffic. 

6.3. 1 Unwanted Traffic Flooding Attacks 

We first simulate the attack scenario where the attackers directly 
flood a victim, but the victim can classify the attack traffic, and 
uses the provided DoS defense mechanism: capabilities in TVA+, 
secure congestion policing feedback in NetFence, and filters in Sto- 
plt, to block the unwanted traffic. 

We desire to simulate attacks in which thousands to millions of 
attackers flood a well provisioned link. However, we are currently 
unable to scale our simulations to support beyond several thousand 
nodes. To address this limitation, we adopt the evaluation approach 
in [47]. We fix the number of nodes, but scale down the bottleneck 



link capacity proportionally to simulate the case where the bottle- 
neck link capacity is fixed, but the number of attackers increases. 

We use a dumb-bell topology in which ten source ASes connect 
to a destination AS via a transit AS. Each source AS has 100 source 
hosts connected to a single access router. The transit AS has two 
routers Rm and Rb r , and the destination AS has one victim desti- 
nation host. The link between R^i and Rt r is the bottleneck link, 
and all other links have sufficient capacity to avoid congestion. We 
vary the bottleneck link capacity from 400Mbps to 50Mbps to sim- 
ulate the scenario where 25K ~ 200K senders (both legitimate and 
malicious) share a lOGbps link. Each sender's fair share bandwidth 
varies from 400Kbps ~ 50Kbps, which is NetFence's targeted op- 
erating region. The propagation delay of each link is 10ms. 

In the simulations, each sender is either a legitimate user or an 
attacker. To stress-test our design, we let each source AS have 
only one legitimate user that repeatedly sends a 20KB file to the 
victim using TCP. We let each attacker send 1Mbps constant-rate 
UDP traffic to the victim. We measure the effectiveness of a DoS 
defense system using two metrics: 1) the average time it takes to 
complete a successful file transfer; and 2) the fraction of successful 
file transfers among the total number of file transfers initiated. We 
set the initial TCP SYN retransmission timeout to 1 second, and 
abort a file transfer if the TCP three-way handshake cannot finish 
after nine retransmissions, or if the entire file transfer cannot finish 
in 200 seconds. We terminate a simulation run when the simulated 
time reaches 4000 seconds. 

For each DoS defense system we study, we simulate the most 
effective DoS flooding attacks malicious nodes can launch. In case 
of an unwanted traffic flooding attack, the most effective flooding 
strategy in NetFence and TVA+ is the request packet flooding at- 
tack. Under this attack, each NetFence sender needs to choose a 
proper priority level for its request packets. We make an attacker 
always select the highest priority level at which the aggregate attack 
traffic can saturate the request channel. A legitimate sender starts 
with the lowest priority level and gradually increases the priority 
level if it cannot obtain valid congestion policing feedback. 

Figure 8 shows the simulation results. The average file transfer 
completion ratio is omitted because all file transfers complete in 
these simulations. As can be seen, Stoplt has the best performance, 
because the attack traffic is blocked near the attack sources by net- 
work filters. TVA+ and NetFence also have a short average file 
transfer time that only increases slightly as the number of simulated 
senders increases. This is because in a request packet flooding at- 
tack, as long as a legitimate sender has one request packet delivered 
to the victim, it can send the rest of the file using regular packets 
that are not affected by the attack traffic. The average file transfer 
time in NetFence is about one second longer than that in TVA+, be- 
cause a legitimate sender will initially send a level-0 request packet 
that cannot pass the bottleneck link due to attackers' request packet 
floods. After one second retransmission backoff, a sender is able 
to retransmit a request packet with sufficiently high priority (level- 
10) to pass the bottleneck link. Attackers cannot further delay le- 
gitimate request packets, because they are not numerous enough to 
congest the request channel at this priority level. 

Figure 8 also shows that FQ alone is an ineffective DoS defense 
mechanism. With FQ, the average file transfer time increases lin- 
early with the number of simulated senders, as each packet must 
compete with the attack traffic for the bottleneck bandwidth. 

These results show that NetFence performs similarly to capability- 
based and filter-based systems when targeted victims can stop the 
attack traffic. A legitimate sender may wait longer in NetFence 
to successfully transmit a request packet than in TVA+ or Stoplt. 
This is because NetFence uses coarse-grained exponential back- 
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Figure 9: Throughput Ratio between legitimate users and attackers 
when receivers fail to suppress the attack traffic. Fairness Index among 
legitimate users is close to 1 in all the simulations. 



off to schedule a request packet's transmission and set its prior- 
ity, while TVA+ uses fine-grained but less scalable per-sender fair 
queuing to schedule a request packet's transmission, and Stoplt en- 
ables a victim to completely block unwanted traffic. 

6. 3. 2 Colluding Attacks 

Next we present our simulation results for regular traffic flooding 
attacks where malicious sender-receiver pairs collude to flood the 
network. Such attacks may also occur if DoS victims fail to identify 
the attack traffic. 

Single Bottleneck: We use a similar topology as in the previous 
experiments (§ 6.3.1) to simulate colluding attacks. In this simu- 
lation topology, the router at the right-hand side of the bottleneck 
link Rbr connects to one destination AS with a victim host and 
nine additional ASes, each having a colluding host (colluder). Each 
source AS has 25% legitimate users and 75% attackers, simulating 
the case where the attackers are numerous but there are still a rea- 
sonable number of legitimate users in each source AS. 

Each legitimate user sends TCP traffic to the victim host. We 
simulate two types of user traffic: 1) long-running TCP, where a le- 
gitimate sender sends a single large file; 2) web-like traffic, where a 
sender sends small files whose size distribution mimics that of web 
traffic. We draw the file size distribution from a mixture of Pareto 
and exponential distributions as in [28], and make the interval be- 
tween two file transfers uniformly distributed between 0. 1 and 0.2 
seconds. The maximum file size is limited to 150KB to make the 
experiments finish within a reasonable amount of time. 

To simulate colluding attacks, we let each attacker send 1Mbps 
UDP traffic to a colluder. The attackers in TVA+ and NetFence 
send regular packets. Colluders in Stoplt do not install filters to stop 
the attack traffic. We simulate each experiment for 4000 seconds. 

When compromised nodes organize into pairs to send attack traf- 
fic, NetFence aims to guarantee each legitimate sender its fair share 
of the bottleneck bandwidth without keeping per-sender queues in 



the core network. We use two metrics to measure a DoS defense 
system's performance under this type of attack: 1) Throughput Ra- 
tio, the ratio between the average throughput of a legitimate user 
and that of an attacker; and 2) Fairness Index among legitimate 
users [11]. Let x t denote a legitimate sender i's throughput, and the 
fairness index is defined as Xi) 2 /(n^xf). The ideal through- 
put ratio is 1, indicating that a legitimate user obtains on average 
the same bottleneck bandwidth as an attacker. The ideal fairness 
index is also 1 , indicating that each legitimate sender has the same 
average throughput. We only measure the fairness index among 
legitimate users because Throughput Ratio has already quantified 
how well a legitimate user performs relatively to an attacker. 

Figure 9 shows the simulation results. The fairness index for all 
systems is close to 1 in all the simulations and is thus not shown 
in the figure. For long-running TCP, NetFence's throughput ratio is 
also close to 1 . This result shows that NetFence provides a legiti- 
mate sender its fair share of bandwidth despite the presence of DoS 
flooding traffic, consistent with the theoretic analysis in § 3.4. For 
the web-like traffic, NetFence's throughput ratio increases gradu- 
ally from 0.3 to close to 1 as the number of simulated senders in- 
creases. The throughput ratio is low when the number of senders is 
small, because a legitimate sender cannot fully utilize its fair share 
bandwidth: each sender has a large fair share of bandwidth, but 
a legitimate sender's web-like traffic has insufficient demand and 
there are gaps between consecutive file transfers. 

FQ and Stoplt perform exactly the same, because in these collud- 
ing attacks, they both resort to per-sender fair queuing to protect a 
legitimate user's traffic. However, unexpectedly, we note that they 
provide legitimate users less throughput than attackers even when 
the user traffic is long-running TCP. By analyzing packet traces, 
we discover that this unfairness is due to the interaction between 
TCP and the DRR algorithm. A TCP sender's queue does not al- 
ways have packets due to TCP's burstiness, but a constant-rate UDP 
sender's queue is always full. When a TCP sender's queue is not 
empty, it shares the bottleneck bandwidth fairly with other attack- 
ers, but when its queue is empty, the attack traffic will use up its 
bandwidth share, leading to a lower throughput for a TCP sender. 

TVA+ has the lowest throughput ratio among all systems in this 
simulation setting, indicating that a small number of colluders can 
significantly impact TVA+'s performance. This is because TVA+ 
uses per-destination fair queuing on the regular packet channel. 
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If this bandwidth is shared by B attackers fairly, each will get a 
b(i+n ) f rac ti° n of the bottleneck capacity. A sender's bottleneck 
bandwidth share in other systems (NetFence, Stoplt, and FQ) is 
G ^ B , and does not depend on the number of colluders Nc ■ In our 
simulations, N c = 9, G = 25% x 1000, and B = 75% x 1000. A 
legitimate TVA+ sender obtains ttft^ of the bottleneck bandwidth, 



while an attacker obtains 



of the bottleneck bandwidth, three 



times higher than a legitimate sender, as shown in Figure 9. 

In these simulations, we also measure the bottleneck link uti- 
lization. The result shows that the utilization is above 90% for 
NetFence, and almost 100% for other systems. NetFence does not 
achieve full link utilization mainly because a router stamps the L* 
feedback for two extra control intervals after congestion has abated, 
as explained in § 4.3.4. 

Multiple Bottlenecks: To evaluate NetFence's performance with 
multiple bottlenecks, we simulate colluding attacks on a parking- 
lot topology that has two bottleneck links in the mon state: Li 
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Figure 10: Sender throughput (Kbps) under regular packet flooding 
attacks in a parking-lot topology with two bottleneck links. The fair 
share rate for each sender is 80Kbps. 
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Figure 11: Average user throughput in face of microscopic on-off at- 
tacks. The user traffic is long-running TCP. There are 100K senders. 
Each sender's fair share bottleneck bandwidth is 100Kbps. 



and L2. 3000 senders are organized into three groups of the same 
size: Group-A senders send through both L\ and L2, Group-B 
senders send through the second link L2, and Group-C senders send 
through the first link L\. Each group contains 75% attackers and 
25% legitimate users. Each attacker sends UDP traffic in regular 
packets at 1Mbps to a colluder, while each user sends long-running 
TCP traffic to a victim. Every simulation terminates at 4000 sec- 
onds in simulated time. 

We simulate three different pairs of bottleneck capacities: 1) 
C Ll = Cl 2 = 160Mbps;2)C Ll = 240Mbps, Cl 2 = 160Mbps; 
and 3) C Ll = 160Mbps, Cl 2 = 240Mbps. In these cases, a 
Group-A sender's max-min fair share bandwidth is 80Kbps, while a 
Group-B or Group-C sender's max-min fair share is either 80Kbps 
or 160Kbps. The simulation results are shown in Figure 10. The 
x-axis shows different simulation cases, and the y-axis is a sender's 
average throughput. Senders in Group-B and Group-C are omitted 
because they each can get at least its fair share bandwidth in all 
the simulations. However, on average a sender in Group-A obtains 
much smaller throughput than its fair share rate when Cl x < C'l 2 - 
This is because traffic from a Group-A sender traverses both bottle- 
neck links L\ and L2. As discussed in § 4.3.5, if a flow traversing 
both Li and L2 switches between the two corresponding rate lim- 
iters frequently, its rate limit may become smaller than its fair share 
bandwidth. This problem does not significantly affect the first two 
cases in which Cl x > Cl 2 , because in these simulations a Group- 
A sender's traffic carries Li's feedback most of the time. As a 
result, the rate limiter (src, Li) for a Group-A sender src is not 
idle most of the time, and it can have a rate limit close to the fair 
share bandwidth of the sender src. 

Figure 10 also shows that a TCP user in Group-A has an even 
lower average throughput than an attacker in Group-A (the right- 
most case). This is because when a TCP flow switches between 
two rate limiters that have very different rate limits, TCP cannot 
catch up with the abrupt rate limit change. 

NetFence's performance with multiple bottleneck links can be 
improved with more complicated designs, as discussed in § 4.3.5 
and Appendix B. We believe the current design is a reasonable 
tradeoff to balance between complexity and performance. 

Strategic Attacks: Attackers may launch sophisticated attacks 
(§ 5.2) than brute-force flooding attacks. We simulate microscopic 
on-off attacks and show that with NetFence, the attack traffic's 
shape does not reduce a legitimate user's throughput. 

The simulation topology is the same as in the previous single- 
bottleneck simulations. All legitimate users send long-running TCP 
traffic, while attackers send on-off UDP traffic. In the on-period 
Ton, an attacker sends at the rate of 1Mbps; in the off-period T D //, 
it does not send any traffic. All attackers synchronize their on- 



periods to create the largest traffic bursts. There are 100K simulated 
senders, each having a fair share bandwidth of at least 100Kbps. 

In these simulations, we use two different values for T on : 0.5s 
and 4s. For each T on , we vary the off-period length T jf from 
1.5s to 100s. Figure 11 shows the simulation results. As we can 
see, the average user throughput is at least a user's fair share rate 
as if attackers were always active (100Kbps), indicating that the 
attack cannot reduce a legitimate user's fair share of bandwidth. As 
the attackers' off-period length increases toward 100s, a legitimate 
user can achieve a throughput close to 400Kbps, indicating that 
long running TCP users can use most of the bottleneck bandwidth 
when the attackers' off-period is long. 

7. DISCUSSION 

Fair Share Bound: When a disproportionally large number (B) 
of attackers attack a narrow link C (e.g., when a million bots at- 
tack a 1Mbps link), the fair share lower bound 0( G ^ B ) achieved 
by NetFence or per-sender fair queuing (e.g., [27]) is small. How- 
ever, this lower bound is still valuable, because without it, a small 
number of attackers can starve legitimate TCP flows on a well- 
provisioned link (e.g., lOGbps). Although this guarantee does not 
prevent large-scale DoS attacks from degrading a user's network 
service, it mitigates the damage of such attacks with a predictable 
fair share, without trusting receivers or requiring the network to 
identify and remove malicious traffic. Other means, like conges- 
tion quota discussed below, can be used to further throttle malicious 
traffic. 

Fqual Cost Multiple Path (ECMP): NetFence assumes that a 
flow's path is relatively stable and the bottleneck links on the path 
do not change rapidly. One practical concern arises as routers may 
split traffic among equal-cost multi-paths for load balancing. Fortu- 
nately, most ECMP implementations in practice (e.g., [12]) would 
assign a flow's packets to the same path to avoid packet reordering. 
Thus, we expect NetFence to work well with ECMP. 

Congestion Quota: If we assume legitimate users have limited 
traffic demand at attack time while attackers aim to persistently 
congest a bottleneck link, we can further weaken a DoS flooding 
attack by imposing a congestion quota, an idea borrowed from re- 
ECN [9]. That is, an access router only allows a host to send a lim- 
ited amount of "congestion traffic" through a bottleneck link within 
a period of time. Congestion traffic can be defined as the traffic that 
passes a rate limiter when its rate limit decreases. With a conges- 
tion quota, if an attacker keeps flooding a link, its traffic through the 
link will be throttled after it consumes its congestion quota. Differ- 
ent from re-ECN, NetFence can impose a per-(sender, bottleneck 
link) congestion quota so that a sender's traffic not traversing any 



link that is under attack will not be incidentally throttled when its 
quota for a link under attack is used up. 

Congestion Detection and Rate Limit Control: For simplicity, 
the current implementation of NetFence uses RED as the conges- 
tion detection algorithm, and the NetFence design uses a closed- 
loop AIMD algorithm to adjust rate limits. RED is not a strictly 
load-based congestion detection algorithm. In the future, we plan 
to explore a load-based congestion detection algorithm that may re- 
act sooner to congestion than RED. Beside AIMD, another option 
to set a sender's rate limit is to let a congested router compute a 
max-min share rate of each sender, and periodically send this rate 
to a sender's access router. We discard this approach because it 
involves per-sender state and has higher computation and message 
overhead than AIMD. 

Control Interval Length: The current NetFence design uses a 
fixed control interval Iu m for all senders. It is difficult to optimize 
the value of Iu m , because a short Ii im cannot accommodate het- 
erogeneous RTTs on the Internet, while a long In-m, slows down 
the convergence of rate limits. A future improvement we plan to 
explore is to have a variety of control interval lengths {Ii im \. A 
sender may signal which length it prefers to use with a few bits in 
its NetFence header. Accordingly, we need to update the rate limit 
adjustment algorithm on an access router and the congestion hys- 
teresis on a congested router to ensure fairness among senders with 
different control interval lengths. 

Convergence Speed: It may take a relatively long time (e.g. , 100s- 
200s) for NetFence to converge to fairness when a sender's rate 
limit is significantly different from its fair share of bottleneck band- 
width. This is because the control interval Iu m is on the order of a 
few seconds (2s in our current implementation), much longer than 
a typical RTT on the Internet. This convergence speed is acceptable 
in the NetFence design, because a rate limiter persists for a much 
longer period of time (i.e., on the order of hours), which prevents 
attackers from constantly taking advantage of the unfairness during 
the convergence by sending strategic on-off traffic. 

TCP and Rate Limiter Interaction: TCP's window size adjust- 
ment may be temporarily out of synchronization with the rate lim- 
iter adjustment. For instance, TCP may just receive its ACK before 
the rate limit is reduced, in which case TCP's sending rate will in- 
crease. This temporary out of synchronization is not a big issue, 
because the rate limit adjustment interval Ium is much larger than 
a typical RTT in the Internet such that the out of synchronization 
may occur at most once among many RTTs. 

8. RELATED WORK 

At the architectural level, NetFence combines the elements of 
capability-based systems [34,46,47] and re-ECN/re-feedback [8,9]. 
In contrast to capability tokens, NetFence's congestion policing 
feedback carries valuable network congestion information. Re- 
ECN/re-feedback is a congestion policing framework that incen- 
tivizes rational senders to honestly report downstream path con- 
gestion. Routers will discard the packets from the senders that 
under-report downstream congestion with high probability before 
they reach the destinations. In contrast, NetFence is a DoS defense 
architecture that uses unspoofable congestion policing feedback to 
scalably and robustly guarantee a sender's fair share of bottleneck 
bandwidth in face of attacks. Attackers cannot send packets with 
false congestion feedback reporting no or low levels of congestion 
to flood a link. Instead, they can at most send packets reporting the 
actual levels of congestion and will not gain more bandwidth than 
honest senders. In addition, DoS victims can use the unspoofable 



feedback as capability tokens to suppress unwanted traffic. ECN- 
nonce [16] robustly signals congestion from the network to a honest 
sender even when a receiver attempts to hide congestion, while Net- 
Fence enables robust congestion signaling from congested routers 
to access routers when both senders and receivers are malicious. 

NetFence's request packet protection mechanism is inspired by 
Portcullis [34] that uses computational puzzles to impose delay on 
senders. Differently, NetFence uses a rate limiting algorithm that 
does not require proof-of-work (PoW) nor a network-wide puzzle 
synchronization mechanism. This algorithm is similar in spirit to 
LazySusan [13] which substitutes resource-based PoW for latency- 
based PoW. Different from LazySusan, NetFence uses a sender's 
waiting time to set its request packet's priority level, and guarantees 
the eventual delivery of a legitimate request packet. 

Several DoS defense systems aim to enable a victim to install 
network filters to stop unwanted traffic [2, 5, 27], or to control who 
can send to it [6]. Unlike them, NetFence does not use per-host 
queues at congested routers to separate legitimate traffic from at- 
tack traffic in case compromised receivers collude with malicious 
senders. Pushback [29] sends hop-by-hop pushback messages from 
a congested router to install per-(incoming interface, destination 
prefix) rate limiters to reduce DoS flooding traffic. NetFence does 
not require hop-by-hop deployment, enables a victim to suppress 
unwanted traffic, and provides per-sender fairness at bottleneck 
links: attackers cannot diffuse their traffic to many destinations to 
gain unfair bandwidth shares. AIP [2] uses trusted host hardware to 
block unwanted attack traffic, while NetFence places policing func- 
tions inside the network and does not require trusted host hardware. 

Speakup [44] and Kill-Bots [22] address application-layer DoS 
attacks, while NetFence addresses network-layer DoS attacks. Sev- 
eral systems use overlay networks [1, 15,23,37,39,41] or middle- 
boxes [10,31] to mitigate DoS attacks against dedicated destina- 
tions. DoS mitigation products on today's market (e.g., [42]) offer 
in-network anomaly detection and attack traffic removal services 
near the victims. Kreibich et al. [25] propose to use packet symme- 
try to detect and remove attack traffic. This body of work requires 
fewer changes to routers, but NetFence can remove attack traffic 
near its origins and protect all destinations on the Internet once de- 
ployed. Moreover, it places the attack traffic identification function 
at the receivers to keep the network open to new applications. 

NetFence's approach to scalability is inspired by CSFQ [40] that 
achieves per-flow fairness without per-flow queues in the core routers 
Differently, NetFence enables DoS victims to suppress attack traf- 
fic, and provides per-sender rather than per-flow fairness. 

9. CONCLUSION 

This paper presents the design and evaluation of NetFence, an 
architecture that places the network at the first line of DoS defense. 
NetFence uses a key mechanism, secure congestion policing feed- 
back, to enable scalable and robust traffic policing inside the net- 
work. Bottleneck routers use the congestion policing feedback to 
signal congestion to access routers, and access routers use it to ro- 
bustly police senders' traffic. In case compromised senders and 
receivers collude in pairs to flood the network, NetFence limits the 
damage of this attack by providing each sender (malicious or legiti- 
mate) its fair share of bottleneck capacity without keeping per-host 
state at bottleneck routers. In case attackers send DoS floods to 
innocent victims, NetFence enables the DoS victims to use the se- 
cure congestion policing feedback as capability tokens to suppress 
unwanted traffic. Using a combination of a Linux implementation, 
simulations, and theoretic analysis, we show that NetFence is an 
effective DoS solution that reduces the amount of state maintained 
by a congested router from per-host [27,47] to per-AS. 
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APPENDIX 

A. CONVERGENCE ANALYSIS 

NetFence uses an AIMD algorithm to adjust a sender's rate limit. 
AIMD has been proven to converge onto efficiency and fairness [11]. 
We first briefly summarize the AIMD results, and then analyze how 
well NetFence converges to fairness and efficiency. 

A.1 AIMD Preliminary 

Consider a simplified fluid model where one link is shared by N 
synchronous flows, all of whom having the same round-trip time 



(RTT), T. Each flow i uses the following rule to update its sending 
rate Xi(t) at time t: 

AI: Xi (t + T) = Xi (t) + a if no congestion 
MD: Xi(t + 5t) = Xi(t) x j3 otherwise 
where a > 0, 1 > /3 > 0, <5t > and St ->■ 0. 

Convergence to efficiency: When the link is under-utilized, AI is 
applied. The aggregate rate continues to increase as X; a;i(£ + 
r ) = Xi + > At some point the link 

will eventually be over-utilized, leading to Xi Xi(t + St) — /3 x 

X; < X; x i(t)> a cul; °f me a gg re g ate rate - MD finally re- 
sults in link under-utilization. This pattern of oscillation around 
full utilization repeats, with the overshoot and undershoot decided 
by Net and p, respectively. Efficiency convergence is achieved if 
a ->• and f3 — >■ 1, 

Convergence to fairness: We measure fairness with Jain's fair- 
ness index [1 1]: (Xi Xi(t)) 2 /N Xi (*)• It is easy to show that 
MD keeps the index unchanged, while AI increases it. So, given 
sufficient number of AI rounds, the fairness keeps increasing, and 
will eventually reach its maximum, 1. At this point, all the flows 
share the link capacity equally. 

A.2 Capacity Share Lower Bound 

Given G good and B bad hosts sharing a bottleneck link of ca- 
pacity C, can NetFence guarantee that each good host obtains its 
fair share 0( G ^_ B ) of the capacity, regardless of the attack strategy 
bad hosts may apply? Here we prove this is indeed true. 

Again we consider a simplified fluid model in Figure 12. There 
are a fixed number of senders and one receiver. To make our anal- 
ysis tractable, we do not consider delay: any rate change at the 
senders immediately takes effect at the bottleneck link. The bad 
hosts can use traffic of any shape to attack. As shown in the figure, 
for any host i, let r\ (t) be its sending rate at time t, and rf be its 
rate limit at its access router, rf is a constant value during a control 
interval [to , to + Iu m ), where to is a particular point in time. Then, 
at the output link of the access router, the egress rate of the host i 
is ri(t) = min(rf (t), rf) due to the rate limit. 

Definition: We say a host has sufficient demand from its appli- 
cation if the host sends fast enough such that its corresponding rate 
limit is not punished by the rate limit AIMD algorithm described in 
§ 4.3.4. 

Definition: During any control interval, given a rate limit rf, 
the rate limit utilization vi = ri/rf, where the host average egress 
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Ti(t)dt. Obviously, we have < Vi < 1. 

Remark: If a sender i's traffic is sent in TCP, Vi depends on 
TCP's congestion control efficiency. Given enough buffers at the 
congested router, this Vi is close to 1 in typical scenarios, because 
NetFence uses a control interval Iu m = 2s for rate limit AIMD, 
which is typically one order of magnitude larger than the end-to- 
end RTT (on average 100~200ms in the Internet). We might have 
a low Vi (e.g., less than 50%) under very low/high bandwidth-delay 
product networks where TCP is inefficient. If a sender i's traffic is 
sent in UDP, i>i depends on the host sending rate rf it) during the 
control interval Iu m . For a source that sends as fast as allowed by 
the rate limiter, we have Vi = 1; for on-off traffic, Vi is decided by 
its duty cycle. 

Lemma: For any host s with sufficient demand, its rate limit 
will eventually be the highest among all the rate limits, i.e., rf = 
maxi(rf). 

Proof. We first show that if all the hosts have sufficient demand, 
they will eventually reach the same rf . This is due to the AIMD 
rule. As we have shown in Section A.l, MD keeps fairness un- 
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Figure 12: A simplified fluid model for NetFence 

changed, but AI increases it. Under NetFence, after an MD, there 
is at least one AI round. This is because if there were only MD 
rounds, the aggregated rate limit would eventually become very 
small, and then the bottleneck link would not be congested, which 
would lead to an AI. Therefore, the rate limit fairness increases as 
time goes on, and eventually all the hosts will have the same rf . 
For a host without sufficient demand, NetFence punishes it by not 
increasing or even decreasing its rate limit. Therefore, the host can 
only get a lower rate limit than one with sufficient demand. □ 

Assumption: Congestion detection signals if and only if aggre- 
gate demand surpasses bottleneck link capacity, i.e., when Xi ~i > 
C. 

Remark: In practice, this is often true for load-based congestion 
detection schemes that signal congestion when the average traffic 
arrival rate exceeds a high-load threshold. Queue-based congestion 
detection (like RED) is more sensitive to traffic dynamics, possi- 
bly signaling congestion due to bursty traffic even if the link aver- 
age utilization in one control interval is low. However, in the case 
of NetFence, according to our simulations, queue-based conges- 
tion detection satisfies this assumption well. This is because each 
sender's traffic is shaped by a rate limiter before it enters a bot- 
tleneck, which significantly limits the peak rate of the aggregate 
incoming traffic. 

Theorem: Given G good and B bad senders sharing a bottle- 
neck link of capacity C, regardless of attack strategy, any good 
sender g with sufficient demand eventually obtains a fair share 
W here p=(l~S) 3 . 

Proof. During a congested control interval, we have C < Xi ?i — 
X,rf < X^ax^rf) = r a g (G + B) and thus r a g > jgg. For 
any uncongested control interval, the rate limit is reduced from the 
peak rate at the congested control interval, so r g > ^ B where 
p — (1 — S) 3 . This particular p value is due to the fact that our 
rate limit adjustment may apply one MD cut to make the link not 
congested and then at most two more extra MD cuts to prevent ma- 
licious senders from hiding Ir feedbacks. Therefore, we have, for 
all the control intervals in the steady state, Tg — VgT'g > ■ D 

Remark: For a multi-homed host h m that uses the bottleneck 
link I via multiple access routers, there is one rate limiter (h m , I) in 
each access router. The host's bandwidth share can thus be higher 
than a single-homed host. This is not a big issue since the number 
of access routers of any multi-homed host is often small (e.g., 2). It 
is also reasonable since the host pays for multiple access links. 

B. ALTERNATIVE DESIGNS TO HANDLE 
MULTIPLE BOTTLENECKS 

Next we present two possible solutions to improve NetFence's 
performance in the multiple bottleneck case as described in § 4.3.5. 

B.l Multi-bottleneck Feedback in One Packet 

A clean solution to handle multiple bottlenecks is to let a single 
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Figure 13: Sender throughput (Kbps) with the same simulation set- 
ting as in Figure 10, but with each packet carrying congestion policing 
feedback from multiple bottleneck links. The fair share rate for each 
sender is 80Kbps. 

packet carry the congestion policing feedback from multiple bottle- 
neck links on a forwarding path. Compared to NetFence's core de- 
sign described in § 4, this solution requires a new NetFence header 
format, a new set of algorithms to stamp and verify feedback, and 
a new regular packet policing algorithm; other parts of NetFence's 
core design do not need to change. 

Multi-bottleneck feedback in a NetFence header: A NetFence 
header may carry the congestion policing feedback from zero or 
more bottleneck links on the packet's forwarding path. A link L 
may stamp one of two types of feedback: or . Their meanings 
are the same as defined in § 4.1. The ith bottleneck's feedback in 
a packet is encoded in two fields: linki and actiorii, whose values 
are the same as defined in § 4.4. 

All the feedback fields in a NetFence header are made unforge- 
able by a single token field, as we will show soon. A NetFence 
header also contains a single timestamp field ts to indicate the 
freshness of all the feedback. 

If a packet does not traverse any bottleneck link, its NetFence 
header will only contain a ts field and a valid token field stamped 
by its access router. We refer to such a NetFence header as contain- 
ing the nop feedback. 

Stamping and verifying feedback: When a packet leaves its ac- 
cess router, the access router stamps the nop feedback. That is, it 
sets ts to its local time and computes the token field as follows: 

token n0 p = MAC Ka (src, dst, ts) (4) 

K a is the same as defined in § 4.4. 

When the packet traverses a bottleneck link L in mon state, the 
bottleneck router inserts its feedback into the NetFence header. It 
inserts Z> if the algorithm in § 4.3.4 determines to do so, or oth- 
erwise. The bottleneck router updates the token field as follows: 

token = MACn ai (src, dst, ts, L, action, token) (5) 

K a i is the same as defined in § 4.4. The computation of the new 
token value covers the old token value to prevent a malicious 
downstream router from tampering the feedback stamped by other 
bottleneck links. 

When an access router verifies a NetFence header, it first checks 
the ts field as described in § 4.4. Then it reconstructs the origi- 
nal token nop using Eq (4), and recomputes the token field using 
Eq (5). When the packet carries feedback from multiple bottle- 
necks, the router will have to apply Eq (5) multiple times to calcu- 
late the final token value. 

Policing regular packets: The multi-bottleneck feedback in a reg- 
ular packet clearly indicates the bottleneck links on the packet's 



Figure 14: Sender throughput (Kbps) with the same simulation set- 
ting as in Figure 10 but also with rate limiter inference. The fair share 
rate for each sender is 80Kbps. 

forwarding path. An access router rate-limits a regular packet with 
all the rate limiters each associated with one on-path bottleneck 
link. If the packet cannot pass any of them, it is discarded. This 
algorithm prevents the rate limit for a flow from changing abruptly, 
and it also ensures that the flow's sending rate is smaller than the 
lowest rate limit for all the on-path bottleneck links. 

Simulation results: Figure 13 shows the simulation results when a 
packet can carry multi-bottleneck feedback. The simulation setting 
is the same as in Figure 10. As can be seen, each sender in Group-A 
can get roughly its fair share of bottleneck bandwidth. 

B.2 Inferring Rate Limiters 

The solution in Appendix B.l requires a new NetFence header 
format, which may be longer than that in the core NetFence design 
described in § 4. An alternative design is that a packet still carries 
the congestion policing feedback from a single bottleneck, but an 
access router can infer the bottleneck links a flow traverses and 
police the flow's packets using all the corresponding rate limiters. 
This solution is compatible with NetFence's core design, because 
each access router can independently deploy it. 

Inferring on-path rate limiters: An access router keeps a per- 
destination-prefix cache that records what bottleneck links are on 
the path towards a particular prefix. Whenever an access router 
receives or feedback from a sender, it adds L into the cache 
entry associated with the destination prefix. L may be removed 
from the cache entry if all the rate limiters for L have been removed 
or the and feedback has not appeared in packets toward the 
destination prefix for a long period of time. 

The number of entries in the inference cache is at most the num- 
ber of prefixes in a full BGP table. An access router also needs 
the prefix list of a full BGP table in order to locate the cache entry 
given the destination IP address in a packet. 

Policing regular packets: Based on the inference cache, an access 
router passes a packet with mon feedback through all the rate lim- 
iters associated with the bottleneck links on the packet's forwarding 
path. If the packet cannot pass any of them, it is discarded. 

The inference cache may be inaccurate: a link L may stay in 
the cache entry of a particular prefix even after it is no longer in 
mon state or on the path toward the prefix. In this case, traffic 
toward the destination prefix may be unnecessarily throttled by the 
rate limiters associated with L. However.this is not a big problem 
because such cases only occur infrequently, and L will be removed 
from the cache entry after traffic toward the destination prefix no 
longer carries the and feedback for some time. 

When a packet with the or feedback passes all the associ- 
ated rate limiters, let (src, Li ow ) denote the one with the smallest 



rate limit. Unlike the algorithm described in § 4.3.3, the access 
router will reset the feedback to Lj ow . 

Inferring rate limits: With the rate limiter inference cache, the 
single-bottleneck feedback in a packet may be used to infer the 
feedback from other bottleneck links on the packet's forwarding 
path. If a packet carries the feedback, other on-path bottle- 
neck links must not be congested at this moment, because other- 
wise the feedback would have been replaced with an L*^ feed- 
back; on the other hand, if a packet carries an feedback, other 
on-path bottleneck links will not be able to stamp their feedbacks, 
and therefore the rate limits for them should be temporarily held 
unchanged. 

With this inference algorithm, the rate limit adjustment algo- 
rithm in § 4.3.4 needs to be updated accordingly. In addition to 
t a and haslncr, a rate limiter (src, L) is also associated with 
three more state variables: haslncr*, isActive, and isActive*. 
haslncr* records whether the rate limiter has seen the L* 1 " feed- 
back with a timestamp newer than t s ; isActive records whether 
the rate limiter has seen any or feedback regardless of the 
timestamp value; and isActive* records whether the rate limiter 
has seen any L*^ or L*^ feedback. At the end of each control in- 
terval Ium, the rate limiter (src, L)'s rate limit ru m is adjusted as 
follows: 

1. If haslncr or haslncr* is true, the rate limiter's through- 
put rtput is compared with \ru m : ru m is increased by A if 

rtput ^ rum, 

2. Otherwise, if isActive is true, rn m is decreased to (1 — 

S)ri im ; 

3. Otherwise, if isActive* is true, ru m is kept unchanged; 

4. Otherwise, ru m is decreased to (1 — 8)ri im . 

Simulation results: Figure 14 shows the simulation results with 
the above rate limiter inference algorithm. The simulation setting 
is the same as in Figure 10 and Figure 13. We can see that with 
this algorithm, TCP senders and attackers in Group-A have roughly 
the same throughput, because the rate limit of a flow no longer 
jumps between significantly different values. The throughput of 
a Group-A sender is improved compared to Figure 10; however, 
it may still be much smaller than its fair share. This is because 
for any Group-A sender src, the rate limits of both rate limiters 
(src, Li) and (src, L2) can only increase when neither bottleneck 
links is congested. This is a fundamental limitation of embedding 
single-bottleneck feedback in a packet: the packet simply cannot 
carry enough congestion information. 

C. PSEUDO CODE 

This section includes the pseudo code of NetFence's key proce- 
dures. 



1: procedure RATE_LIMITER.RATE_LIMIT_REQUEST_PACKET(pfct) 

2: hdr <s— get_netfence_header(pfct) 

3: if hdr. priority == then 

4: return PASS 

5: end if 

6: ts now <— get_current_time() 

7: token now <— m_token in i t + (ts n0 w — rn_ts ini t) * 
m_rate in it 

S- inhpn J— 9 hdr - priority -1 

o. LurxjCi ^remove ~ ^ 

9: if token remove > token now then 

10: return DROP 

11: end if 

12: if token now > m_depthi n it then 

13: token„ ow <— m_depthi n u 

14: end if 

1 5 ■ TTL t ohjGTl'j.'t'i'it ^ tokGTLfionj toIx/GTlf^ 777 ove 

16: if mjtokeninu < then 

17: mjtokeninit <— 

18: end if 

19. 777; tSiiiit i tSnow 

20: return PASS 

21: end procedure 

Figure 15: NetFence request packet rate limiting pseudo-code. m_* 
are member variables of the rate limiter. 



1: procedure RATE_LIMITER.RATE_LIMIT_REGULAR_PACKET(pfct) 

2: rate_limiter.update_outgoing_rate(p/rf. len) 

3: r <— rate_limiter.cache_packet(pfct) 

4: if r == CACHED then 

5: return CACHED 

6: else if r == DROP then 

7: return DROP 

8: end if 

9: return PASS 

10: end procedure 

11: procedure rate_limiter.cache_packet(p/cO 

12: ts now <— get_current_time() 

13: if m_cache. size_pkts()== then 

14: if (ts now - m_ts_depart regular ) * m_rate regular > 
pkt.len * 8 then 

15. m ts depart regular ^ tSjiow 

16: return PASS 

17: else if caching_delay_too_long(pfct) then 

18: return DROP 

19: end if 

20: end if 

21: m_cache.enque(pkt) 

22: if m_mc/ie.size_pkts()== 1 then 

23 : rate_limiter. schedule_next_unleash() 

24: end if 

25: return CACHED 

26: end procedure 

27: procedure rate_limiter.unleash_packet() 

28: if m_cac/ie.size_pkts()== then 

29: return NULL 

30: end if 

31: pkt <— m_cac/ie.deque() 

32: m_ts_depart regu iar get_current_time() 

33: if m_eac/ie.size_pkts()> then 

34: rate_limiter.schedule_next_unleash() 

35: end if 

36: return pkt 

37: end procedure 



Figure 16: NetFence regular packet rate limiting pseudo-code. m_* 
are member variables of the rate limiter. 
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procedure RATE_LIMITER.UPDATE_STATUS(pfct) 
hdr <— get_netfence_header(pfct) 
tSpkt <— recover_timestamp_from_packet(/idr) 
if tSpkt > m_t a then 

if hdr. action == I NCR then 

m_haslncr <— TRUE 
end if 
end if 
end procedure 

procedure rate_limiter. adjust_rate_limit( ) 
action <- KEEP 
rate i d m_rate regu i ar 
if m_haslncr then 

if rate_limiter.get_outgoing_rate() > 

m_rate regu i ar /2 then 

action <- INCREASE 
end if 

else 

action <- DECREASE 
end if 

it action == INCREASE then 

else if action == DECREASE then 

m_rate reg uiar <- rn_rate regu i ar * (1 — 5) 
end if 

if action ^ KEEP then 

rate_limiter.update_packet_cache(TOte id) 
end if 

mjiaslncr <— FALSE 
m_t s <— get_current_time_in_seconds() 
rate_limiter.schedule_next_rate_adjustment() 
end procedure 



Figure 17: NetFence rate limit adjustment pseudo-code, 
member variables of the rate limiter. 
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procedure ROUTER. FORWARD_PACKET(pfct) 
q <— router.find_output_queue(pfci) 
if q == NULL then 

discard_packet(pfct) 

return 
end if 

if is_legacy_packet(pfct) then 

q.enqae(pkt) 

return 
end if 

if router.is_from_my_hosts(pfci) then 
r -s— router.rate_limit_packet(pfci) 
if r == PASS then 

q.enque(pkt) 
else if r == DROP then 

discard_packet(pfci) 
end if 
end if 
end procedure 

procedure ROUTER. RATE_LIMIT_PACKET(pfci) 
hdr -s— get_netfence_header(pfct) 
if hdr. mac —— or not router.mac_is_valid(pfct) then 
r <— router.get_init_pkt_rate_limiter(pfct) 
if r.rate_limit_init_packet(pfct)== DROP then 

return DROP 
end if 

else 

if hdr.linkid ^ then 

r <— router.get_regular_pkt_rate_limiter(pfct) 

r.update_status(pfci) 

x 4— r.rate_limit_regular_packet(pfct) 

if a; == DROP then 

return DROP 
else if a; == CACHED then 

return CACHED 
end if 
end if 
end if 

router. update_packet(pfci) 
return PASS 
end procedure 

Figure 18: NetFence packet forwarding pseudo-code. 



procedure queue. deque_regular_packet( ) 

pkt queue.pick_next_regular_packet_to_deque() 

q <— queue. get_regular_queue(pfct) 

if q.ts m on > then 

queue.update_netfence_header(pfct) 

end if 

return pkt 
end procedure 

procedure queue. check_packet_loss( ) 

tSntrm <— get_current_time() 

qset <— queue.get_regular_queue_set() 

for q in qset do 

dr g.get_drop_rate()/g.get_deque_rate() 
q.drop_rate <— q.dropjrate * 0.9 + dr * 0.1 
if q. drop _r ate > p t h then 

O.tSmon ^ tS n0 w 
else if q .tSmon ^ ail(l tSnow Q.tSmon T^ecover 

then 

Q .tSmon 4 1 

end if 
end for 
end procedure 



Figure 19: Pseudo-code showing how a bottleneck router may update 
a packet's congestion policing feedback. 



